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ABSTRACT 

In a collaborative effort, the Federal Communications 
Commission- (FCC) and the Document Design Project conducted an 
evaluation of marine, radio rules for recreational boaters that had 
been rewritten in plain English by JF££ personnel. The prevised rules 
were evaluated by 53 experienced boaters and 52 inexperienced 
boaters, who t?ere given either the old rules or the new ones and then 
tested^f or .their comprehension of the rules. EacH question on the 
test required the subject to find the appropriate rule and apply its 
provisions. The subjects also noted ,the,.time it took them to complete 
each question and rated the overall /difficulty they had in using the 
rules. Results showed «n| -subjects;*" using the revised rules were^ 
significantly better atTidentif ying the proper rules, t.ook less time 
to answer questions aboijt the rules,, and rated them as easier to use 
than did subjects using the old rules. (A copy of the comprehension 
test is appended.) (FL) ' 
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ABSTRACT 

In a collaborative effort, the Federal Communications 
Commis.si.on- (FCC) and the Document Design Project conducted an 
evaluation of marine, radio rules for recreational boaters that had 
been rewritten in plain English by J£££ personnel. The ^revised rules 
were evaluated by 53 experienced boaters and 52 inexperienced 
boaters, who Here given either the old rules or the new ones and then 
tested for their comprehension of the- rules 1 . EacH question on the 
test required the subject to find the appropriate rule and apply its 
provisions. The subjects also noted the,. time it took them to complete 
each question and rated the overall/difiiculty they had in using the 
rules. Results showed tthfo -subjects;" using the revised rules were^ 
significantly .better atTTdentif yiirg the proper rules, took less time 
to answer questions aboijt the rules,, and rated them as easier to use 
than did subjects using the old rules. (A copy of the comprehension 
test is appended.) (FL) 
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■ , THE EV^C^ATION OF * A PUBLIC DOCUMENT^: 
Th^ase of FCC' 1 s Marine Radio. Rules for 
Recreational Boaters" 

* ( 

Abstract 

The Federal Communications Commission 
revised its marine radio rules for recrea- 
tional boaters by->writing it in plain 
English and by incorporating various docu- 
ment design prinoipyles. The revised rules 
were evaluated by d2 X 2 factorial ANOVA: 
One hundred five subjects (formed into ex- 
perienced and non-experienced boater 
groups)''were compared' on how well they 
used and unfle'rstood the original and re- 
vised rules. Groups using the revised 
rules were significantly better in iden- 
tifying^ the proper -rules. IiViahswering 
questions about the rules, they took less 
time to answer questions. " They also rated 
the revised rules as easier to use. These 
results and the implications of cpnducting 
empirical evaluations of public document? * ' 
are discussed. 



Introduction 

Background . 

The Federal Communications Commission (FCC)' is the agency 
concerned with regulating all non-government wire and radio ■ 
communications. .The FCC's sphere of infl uence* and- responsibility 
is immense: it oversees^all public and commercial TV and radio 
ytoadcasts; it regulates? interstate telephone rates; and it 

4 

supervises all two-way r^dio services used by police and fire 

departments and used in aviation and marine operations. -In 

* ' . ■ J *• 

carrying out its mandate,, the FCC publishes, monitors, and enforce 

■ % ■ 



< many thousand's of rules and regulations aimed at audiences ranging 
from lawyers and engineers \o the truck ^driver buying a CB radio. 
And, in common with other government agencies, most' of the t rules, 
and regulations issued by tY\e FCC are hard to read and understand. 

t 

The FCC has been in the forefront* of^Federal agencies to take 
serious steps> to simplify some of i,ts rules and regulations. The 
* rules governing citizens band .radro) service (CB radios) were 
rewritten into plain English in 1978. The lawyers who rewrote 

*» • 

these ^ules completely reorganized them. They replaced tec)firKical 

M m \ 

words and legal phrases with common^terms . They used informative 

headings and tables/ instead of dense prose. White space was 
liberally* used betv/een paragraphs and in the margin^. The CB rales 
were widely recognized throughout the Federal establishment as a \ 
-piodel for others in the government to follow in order to make a 

public rules and regulations easier to read an3 understand. 

) 

However, no empirical evidence was aver offered- to support the 
claim that the 'revised rules are comprehensible or that they are an 
improvement over the original version. 

I - ' ■ 

In a second major effort, the FCC revised Part '83 of Chapter I 

of Title 47 of the Code of Fe-deral Regulations, Which de-als with 

t 

*. 

FCC's marine radio, rules fq,r recreational boaters. The FCC 1 s goal' 

was to produce a low cost booklet of rules that the 30*0, 000 / 

* * 

recreational boaters who have two-way marine radios on their boats 

could use and understand.. The writers extracted information that 

* 
i 

specifically relates to -recreatipnaT boating from a ipuch larger set 

I * 
of rules which also covered radio communication among largo 



commercial ships and vessels. In revising the marine radio rules 
into, an 11-page booklet, they followed document design 'principles 
that are-.similar to the ones that' the authors of -CB r^dio-rules had 
jjsed^. This time around, the FQC wanted to know whether the revised 
rules, in fact, 'are usable, readable, and easy to understand, and 

/ 

whether they, are any better than the original rules. 

FCC .staff directly involved in revising the -rules contacted 
the Document Design Project { DDP) of the American Institutes for 
Research (AIR) for assistance in developing and conducting an * 
empirical evaluation of the revised rules. The DDP agreed to 
provide technical assistance to FCC at no cost under the provisions 
of AIR'p contract with the National Institute" of Eduction. This 

contract permits AIR to support selected document design research 

r ' , ■ • - 

and development efforts in Federal . agencies as" cases of putting 

research into practice. 



Purpose < ~ 



• Tbe evaluation was a collab*tative effort between the FCC and 
the Document Design Project. Each group had different ends that t 
they wanted the evaluation to achieve, although these ends were 
complementary* . ' ' > 



One major?and obvious goal of the FCC was to determine whether 
• * 

all the effort that went into rewriting the mariYie radio rules, 
while continuing to fulfill other day-to-day responsibilities, was 
worth Xt. Did revising- the rules into plain English and did-, using 



recQmmended document design principles result in an understandable 
usable document?- Was it any improvement over the existing rules?' 

The. second goal of the FCC's desighers was more implicit and 

political/ They wanted to demonstrate to skeptical superiors that 

the resources expended for improving documents are worthwhile 

investments. It takes considerable money, time, and staff for 

government agencies to formulate, develop, and approve rules and 

regulations. Yet, iY .existing staff can be trained to design 

bette-r public 'documents, and if the same staff can evaluate these 

documents and demonstrate that the public understands and is able 

to use them, then the investment costs become defensible. 

% 

For its part, the Document Design Project bf AIR had both 
scientific and practical purposes .in« supporting this evaluation. 
As document design scientists, we wanted empirical verification of 
the value of ^widely recommended document design principles,, that 
were used in the revised marine radio rules. R&r all of the* 
recommendations that are offered for making document's clear and 
useful by countless writing and design "experts," the fact is that 
there is minimal i support for their assertions in the Research 
literature. This evaluation would help fill a gap in our knowledge 
about the effects of document -des igft principles; on document users. 

At a more applied level, wd wanted to v test the ability* of , 

non-researqh trained government employees to^design # and execute an 

♦ 

empirical evaluation with minimal assistance from experienced 1 
evaluator#. One of^ our more sobering observations in the 2*1/2 
years of providing^ technical assistance to government agencies is 



. * , « 

♦that the .vast majority of regulations, forms^ and rules issued 
annually by the government, even those tHat directly affect the 
well being of -millions of people, are never tested beforehand oh * 
the intended audience. Sometimes this is because the Federal 
agency misunderstands or is unaware of the concept of evaluation; 
in other cases, the Federal agency believes evolution is too ' 
complicated or too expensive to do,. Because* we firmly believe, that 
critical public documents must be evaluated if they, are to serve 
people, and because the resources available to support document 
evaluation are likely to remain limited, we wer*e interested in the 
degree to which typical, non-research trained document designers 

can plan and execute useful , if not scientifically impeccable, 

■» * • 

empirical evaluations. ' 

' P 

In the> remainder of this report we will describe the 
evaluation along the lines of a conventional research report. 
However, unlike a* typical research report, we will intersperse 
•remarks that relate to practical issues of ge tting 'useful > ~ 
evaluations done in the real world of document desigrf. 

The Evaluation Plan 

9 • The evaluation paradigm we used was a straightforward 
comparative, experimental design. Basically, the evaluation" 
experiment consisted of presenting the subjects, which iy this case 
were experienced and inexperienced recreational boaters, with 
•either the , new or old marine radio rules. We then asked them 13> 
teat cfuestions -that concerned information they had read in the 



eric s 
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9 , 

rules-, 'We compared the* performance of the subjects on this test in- 

order to detect any differences between the new and old rules, 
* . * ■ 

With this overview in mind, we nc5w' describe the elements of the 

evaluation plan in mor^e detail. 

The Sample of User s * % ' 

The proper subjects for this evaluation are the -kinds 'of 
people who would be expected, to ufse the marifik radi^o r.ules while 
boating. This breaks tfown' into two distinct audiences: ^ 
experienced recreational boaters and inexperienced boaters. 

'Moreover, given the constraints in resources and time,- the subjects 

* »<- 
had to be drawn from populations convenient to FCC 1 s document 

designers. „ * t 

Experienced boaters were rfeeded because 'the revised marine 
radio rules will replace the existing rules and we wanted to test 
whether 'fafniliarity with the old. rules helps -or hinders 
understanding the new rules. Inexperienced boaters were needed 
because new owners of recreational boat.s also will b£ expe'cted to 
use the'new rule^. We -cou^Ld not presume that all boaters have* 
prior experience with eifher marine radios or the existing rules so 
we wanted to test brow well inexperienced boaters understand the new* 
FCC rules. 

The experienced subjects were 53 male'members of a power 
Squadron in the Washington, t).C* metropolitan area who volunteered 
to .participate in the evaluation. The -Power Squadron is an, 
organization of^boating enthusiasts who own their own recreational 

e 9 



boats and have extensive boating and marine radio experience. We 

refer .to these subjects as the "Power" group. 

' " ' (f . 

The inexperienced subjects were. 52 people recently employed by. 

the FCC Licensing Division at Gettysburg, Pennyslvania . This group 

^as predominantly female < 4'8 out of 52). None had any experience 
* • i 

' with marine radios or with recreational boating. For convenience, 
we refer to these subjects as the\ v " Gettysburg", -gjroup. * 

' The sample thus consisted of 105 intended and potential users 
Of. the revised marine, radio rules. The sample admittedly failed to 

» N 4 

meet the established norms of scientific research — e.g., the 
subjects weSre not randomly dr^wn from their 'respective populations, 
tfiere was an uneven -distribution of males and females. 
Nevertheless,, we believe that the makeup of these groups was 
sufficiently representative of the potential audiences excepted ta 
use the*marine radio rules. /The sample was suitable for meeting, 
the essential requirements *of the evaluation. 

The Materials " m * 

. Old Rules 5 The "old" rules consisted of 49 pages that . 
contained information related to the use of marine radios%in 
recreational boats. These pages were taken from a larger Volume of,. 
Part 83 of the original regulations. An index was specifically 
created to help the subjects locate particular rulek' in the 19 'page 
document. A sample page of the odd rules, written in typical 
"bureaucratese," is shown ,on the next page. 



83.115 Retention of radio station logs. 



• (a) All .station logs which are required under those provisions of this part 

1 pertaining to the particular classes of stations subject to this part 'shall be 

retained by the* licensee for a. period of *one year f rota date of _ entry and for such • > 
additional periods as.r$qulre4 by the following Subparagraphs: * 
(It) Station logs involving, communlcacions incident to a distress 
* or disacer shall be retained by the station licensee for a 
m period of 3 y ears from dace of entry;* 

(2) Station logs which Include entries ef communications incident 
to or involved in an investigation 'by che Commission and con- 
cerning which the station licensee has be'en notified shall be' 

* retained cy the station licensee until such licensee is speci- 

fically authorized in writing by che Commission to destroy them; 

(3) Station logs- incident co or involved in any claim or complaint 
of which the station ''licensee has notice shall be* retained by* 

• such licensee until such claim or complaint has been fully * 
satisfied or until the same has -been barred by statute limiting . 
the time for the filing of suits upon such claims. 

Note: See Part 42 of this chapter concerning preser-r 
vation of records of .common carriers. v 
Tb) Station logs shall be made available to an authorized representative 

of] the Commission upon request. ' 
(c) Ship station logs shall be fully completed -at the end of each voyage 
and before the operator (s) (or ocHer person(s) responsible' under the/ applicable 
provisions-of chis. part ) leave-the ship. Unless otherwise authorized by the 
applicable .provisions of chis part, the radio log currently in use shall be 
kept by che licensed operator (s) of che s^jacion and during' use shall be located 
at che principal radio%operaclng room pf che vessel. Ac che conclusion of eaclL*. 
ocean voyage terminating at a pore of che Uniced. Scaces (includes Puerco Rico, 
and Virgin Islands), che original radio log (or a dupllcace thereof) 1 dating from 
f the last departure of the vessel from a u\S*-port shall t De retained under proper 
custody on board che vessel for a suffcienc period of time (not more chan 24 
hours) co be available for inspection by duly auchorized represencatives of the 
Commission. After rete^rftion on board the vessef as herein scipulaced, che ori- 
1 * ginal log (and che duplicace log if provided) may be filed at an escablished 
shore office of che scacion licensee,' and shall be recained as stipulaced by 
paragraph (a) of this seccion. . » • 

Mote: Duplicace logs are not required by che 
provisions of this paragraph, unless che 
original log is removed 4 prior co opportunicy 
for official inspeccion. , 
* Xd) Logs of ships of che Uniced Scaces concaining encries required co be 
made by reason of the Greac Lakea Agreement or 83.368(c) of this part shall be 
k#pt ac che principal radiotelephone operating location while the vessel is 
being navigated. All entries in their original form required by said agreement 
"or 83.368(c) shall be retained on board the vessel for a period od not less chan 
one month from che date of entry. After retention on board che vessel as herein 
stipulated, .the entries shall be filed at a plate where they vi}l be commission 
upon requent, and shall fc>e retained as stipulated by paragraph, (a) of chis 
seccion. 

^ Example of "Old" Marine Radio Rule* 



I ^ New Rules . The revised rules that were developed by the FCC 

* ^ ? 
* * . / ' ■ * / 

consisted of ah 11-page booklet organized into 22 rules. The new 

-rule^ covered the same conterit areas as. the old rules, but were 

* completely reorganized, redesigned, and rewritten. A sample rule 

of the new version is shown below. Note that paragraph 83.115 from 

the old rules and ruLe'15 from the new, rvu^es cover the sa^e issues 

(keeping rac^io logs) .- * 



VHF Marine Rule 15 

Do I have to ke*p a radio log? 

(a) You must keep a radio *og A**adio log « 
a book in'which you keep mformatipn about 
your radio the radio log must.be neat and 
orderly Each page of the log must be num- 
bered signed by the operator and show the 
name and call sign of your boat. You must 
keep your radio fog for at least one year after 
the day of the last entry in i^e log : 

(b) You just make the following entnes in \ 
your radio log 

• 0} Each distress (MAYDAY) message you 
send or hear 

{2} Each urgency (PAN PAN) or safety 
{SECURITY) message yoti send and 

(3) The installation and servicing of your 
radio 

(c\ For more information on distress mes- 
sages urgency messages and safety mes- 
sages see VHF Marine Rule 19 » 



Sample Rul^from FCC's Revised Marine Radio Rules 
for RecreationafBoaters^ / 



Performance Test . All , sub jects were administered a test 
designed to measure their comprehension of the lfu-les. The test 
cortsistecj of 13 questions about various aspects of marine radio 
operations . ^ The questions represented important and frequently 
occurring situations where the regulations ought to be consulted, 

'Siace sub jectsr did *not h$ve tg^z&ad all sections of t\\$ re&ulatidns 

\ f 
to be able to answer the questions, the test was not exhaustive, • 

•However, we believe the test provided a *wide range of typical 

situations/ } As such, the experimental results could be generalized 



ERIC 
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..to any other 'sample of questions that might have been chosen, 
♦Sample • te*st question i^ sh9wn l^e-^ow. ' ■* • < 



^ Thomas Mriler is licensed' to operate' ^marine ' 

radio aboafcd his reCreatdpna], boat. He then 
; ! - sel'ls his ooat to a neighbor and tells him tfiat 

. the ship^/t^tion license goes with the* boat, * . • ' „ 
> m . 4 ^ Is Mr,'' Miller cptrect? t 

Yes or No (circle one)' , * ' ^ * - 

Why-or\^Why^not4 ' . 

^ Each test question required' subje'cts\to first find the' 
.appropriate rule pertaining to the question and then apply its 
provisions" to the question. Several qu^sti^ons had more than one 
part. .In~all, there were 20 possible "points." Thus, a subject 
wfio answered all the questions completely and accurately scored 20 

m 

on the test; those who answered fewer questiQns or answered them 
less accurately received lower scores. The subjects also noted the 
time when they began each question £nd when they finished..*' The * 
complete test is presented in the appendix. 



' Design and Analysis A 

.J$ The design of the experiment required all subjects to answer ■ 
the same .questions . iEfce 2X2 factorial experimental 'desigrf ( 2 ' 
levels of experience- and 2 forms of rules) i r s depict^ in Figure 1, 



, 13 
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Gettysburg . 
(Inexpeh'enced) 



Power' 

(Experienced) 



New Rules 


Old Rules 



















( 



Figure 1. Experimental design used to evaluate FCC's 
revised rules for recreational boaters. f\ ) y 
* 

The Gettysburg and 'Power groups were tested separately, at * • 
different times and locations. Each subject was randomly assigned 
to either the new or ol.d marine* Yadio. rules and given a copyjof the 
13-questj.on test. The subjects answered each test question in 
sequence; they used 'the rules to ; note the particular rul£ that was 
appropriate for each question artd also to answer the questions 
themselves." E&cti subject also recorded the time taken to complete 
each of the 13 questions. This was done by us-ing ^wall-clock time, 
rounded off c \£o the, nearest minute. Finally, suh^e<?ts rated the 

oyej^ll difficulty they had in using the marine r^dio rulesi ^ 

** * 

In analyzing these data, we fvrtt described and summarized the 
results obtained for each of these measures. We then applied the* 
analysis of variance * ( ANGVA) /to determine the J direction and degree 
of difference in the measures for each group. Finally, we ' ' 
conducted a " secondary" analysis ,that examined the patterns of 
errors made by each' of the groups. ' 



erJc 
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The -rationale for these analysed .rests oft ^the relationships 
among the performance measures collected and the underlying concept 
that they aire, assumed to reflects Tn .-this case, we assumed .that 
the d^'penrde^t measures — number bf answers correct, number of rules 
correct, time 'per question, and ratings of diff ictaj^ty — were valid 
reflections ol the readabiJL i ty , > unders tandabijL i ty , 2nd usefulness 
of the marine radio^rules. 'The statistical tests we conducted were 
merely "checks" to determine the probability that* any result cduld 

have occurred by chance. ' • 

I - . 

• ■ f 

Procedures * 
— ■ * 

* 

The. subjects were rurf* in large groups. After they were 
seated, each ^person was givfen a packet ox material at random. The 
packet contained"^ i ther the old or new versions of* 'the rules and 

the test. . " / 

m * > 

The FCC admin istrator explained the purpose of the study. The 
group was instructed fc'£T£>egin with question #1 and to note the 
particular rule that *&ppl ied ,• answer • the question by reading the 
appropriate rule, ^nc^enter the t f ime taken' to complete the question 
to the nearest minute. A large wall clock visible to the entire 
group was used for timing. Following question 413, the subjects 
rated the version of rules they used for difficulty. The subjects 
worked throug-h the questions at their own pace. When the e^rtrtre 
grgup had finished, the FCC administrator collected the^dles and 
tests and answered any questions* 
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s~«~^ Result ^ 



As we notetf .earlier, the purpose oi^-the evaluation was to 
ascertain >h ether the .revised marine radio rules are comprehensible 
a-nd usable) and <*to ^ test whether they are an improvement over the 
original rules.. ^ To help us make these determinations, the 
evaluation generated four different scores for each subject: 

- ^\ * ' 

• The number of answers correct (out of* 20) — a measure of, 
comprehensibili ty. ■ ' ' 

• The nu*nber%f* rules correctly identified (out of 13) — a 
measure- of both usability and comprehensibili ty . 

i • 

• The time needed to answer each questibn-^-a measure of 
% -usability ... . v • ^ , 

• An overall rating of difficulty in using th£ rulesr- a 
measure of both usability and comprehensibility . 

_) 

Table 1* on th£ next page presents the summary statistics of 
the re suits /obtained from the experiment. will explain Table 1, 

before proceeding to the specific analyses of each dependent 
measure. • 

Section "*A" of Table 1 summarizes the number of test questions 
y W , * 

answered correctly. The first two 'columns 'illustrate the total 

number* of possible correct answers and the actual number of correct 

answer^ for'each group of subjects. For example, tjie first line of 
• * 

Section "A'i shows that out of the 500 possible correct answers for 
the Gettysburg-Old group (i.e., *0 scorable test points X 25 
subjects = 500)% only 30)l actually were answered correctly.' The 



TABLE 1 












Ba|ic Descriptive Infopiation 










A: Test Items Answered Correctly 










* Subject 
Group 




Number of 
. Possible Correct 
Answers 


Number^ 
Answered 
Correctly 


Mean 4 

Number 

Correct 


Standard 
Deviation, 


Gettysburg - Old 


(N=25) 1 


500' 


304 


8 54 


„ 321 ** 


f .Power -Old * y 


(N=25) 


" 500 


422 


1 2 78 


3 78 


Gettysburg - New 


(N=27) 


540 


537 


17 26 


261 


Power - New 


(N=28) 


560 


• 549 


1645 


3 53 


Combined Old 


(N=50}~ 


\ 

1,000 . 


726 (71%) 


10 66 




Comb^ied New 


<N=55) 


1,100 


1,086 (99%) 


16 85 . , 




g: Rules Identified Correctly * \ 


Gettysburg - Old 


<N=25) 


325 


" 203 


6 60 


2 36 


s Power - Old 


{N=25K 


325 , 


238 


6 96 


3 23 


Gettysburg - New 


(N=?7) 


35> 


346 ( v 


11 15 


1 68 


'Power - New 


(N=28) ^ 


364 


336 


9 80 


3 40 


Combined Old 


(N=50) 


650 ' 


441 (67%) 


6 78 




Combined New 


(N*55) 


715 


682 (95%) 


10 47 




C: Tme Per Question (Minutes) 


i 








• ■* 




* Number of 
' Quest tons * 
^nswered 


MearvMtnutes 
to Answer 
Each Question 


Standard f 
Deviation 

« 




Gettysburg - Old 


(N=25> 


226 • 


3 51 


1 47 




Power - Old 


(N=25) 


225 


2 43 


1 22 




Gettysburg - New 


(N-27) 


322 


1 73 


0 63 


Power - New 


(N=28) 


319 


1 50 


0 58 





Combined 644^^jN=50) 
Combine^ -New (N=55) 



478 
641 



2 97 
1.62 



D: Difficult Ratings (5=Hard, 1=easy) 



i i — u - 



Mean 

Difficulty 

Rating 



Gettysburg - Old 


<fN=25) 


4 57 


i 






4 


Power - Old 


(^=25) 


4 22 






• 


Gettysburg - New" 


(N^27) 


2 00 










Power New* 


(N-28) 


i 77 










Combined Old 


(N=50) 


4 39 










Combined New 


(N=55> 


*. 1 88 











14 



17 



third column sfrows the mean (average) numbfer of questions answered 
correct!^ per subject. The fourth column presents the standard - 
deviation for each grpup^^re ma jor comparision of interest, the 
performance of subjects using the new rules vs, ' those using the ^ 
old i^iles, is shown by the "Combined Wew and Old" headings . 

* Section n B" of Table 1 summarizes the number of rules 
correctly identified by the subjects and contains the same- 
information as Section "A." ■ * 

Section M C" of Table 1 summarizes the amount of time (in ' 
minutes) takento answer the test items. The first column shows 
the total nujnber of questions answered by^ each group. The second, 
column presents the mean (average) number of minutes taken to 
answer each question for each group. 

\ 

Section "' D" presents the average rating of difficulty each ' 
group gave to the version of rules^-assigned to them (5=hard, 
l=eas^). » ♦ * 



On the >basis of Table 1 information alone, it is clear'that 
there are performance differences between subjects using the old 
rules and those using the new rules. % We discuss the statistical 
tests used to evaluate the significance of the'se differences next, 



Questions Answered 'Correctly 



/ 



Table 1 illustrates that subjects using the new rules answered 
more test questions correctly. Those using the new fules scored "an 
averag« of 16.85 points (out of 20)< compared to 10.66 points for 

15 f 

18 



V 



those usirftj the old^rules. Figure 2 graphically displays the 
differences in performance for eac^/group. It is clear - that both 

4 



20 r- 




New Rule 



Figure 2 The average number of questions answered 
"fcorrectly^out of 20} by using old and new recreational 
boating rules. 



experienced a<id inexperienced boaters answered more items 
corrrectly with the new rjjles. As might be expected, the 
'experienced boaters performed better with the old rules (Power-Old) 
than inexperienced boaters (Gettysburg-Old) , but this difference 
'disappeared with the use of % the. new # ruj.es . 

4 t 

Results from the 'analysis of variance (ANOVA) indicated that: ' 

* * ' 

, • The difference between the old and>new versions of the 
rules was statistically significant, F(l,101) - 91.85, 
p<.01. ' ' " 

• > The difference 'between tjje two groups (Power and ' 
Gettysburg ) % was statistically significant, £(1,101) = 7.03,, 

RX.oi, r 



• The interaction of groups and rules wasi significant, 
F(l,101) = 15.29, p<.01. ^ \ ' 



Our interpretation of these results is straightforward. tfhe 



10 



1 r 



0 



revised rules are significantly superior to the old rules in terms 
of answering test questions correctly. Moreover, inexperienced 
boaters are just as successful as experienced bo'a^rs 4 ( in 'fact* a 
little better) in their ability to use the new rules' to answer the 
test questions, <>We conclude fro/n this that the new rules are 
comprehensible . 



Rules Identified Correctly 



\ 



Figure 3 shows the average number of rules each grqup^ 
identified correctly when using the new or old rules. The subjects 



using the newVules correctly identified, an average 



C 



• i 

c 
o 

> 

I 

3 

J 

3 

A 

o 



13 



10 



* , 5 



E 



I' 




) 



Power 



o~ — o Gettysburg' 



t Old Rules 



New Rules 



Figure 3 The average number 6f rules identified 
correctly-Jflut of 13} by usin^old and new recreational 
boating rules. 4 



of 10.47 rules (out o£ 13) and thosfe using the old rules an. average, 
of 6.78. This difference's statistically significant, £(1,101) - 
46,85, p<.01. ( Thus both experienced and inexperienced boaters were 

better able to locate aTfid identify the correct rules needed for 

f * 

answering questions »wi th the new rules. The groups did nob differ 
in the*r>-abili ty to find the porrect rule, and the . interaction was 
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not significant. A reasonable interpretation of s this finding is, 
that the new rules 'are more comprehensible and' easier to use . 



y 



Time to Answer, 



Figure" 4 shows the average amount of time' taken by the ^two 
groups to answer each test question using the old and new rules. 
Included in this is the time taken to read the questions, locate 
'the proper rule, read the rule, ♦and answer the questions. 



c 
2 



a 
E 




Power 



o — -o Gettysburg 



Old Rules 



New Rules 



Figure 4 The average tim^ taken to answer each £ 
test question using old and nev$ rules 

Subjects using the .new rules took significantly less time to ariswer 
test c^jestiQns (1.62 minutes on tlrie average vs. 2i97 minutes). In 
addition, The Power grfeup was significantly faster overall and the 
" interaction was * s ign i f roan t (the Gettysburg group improved .more 
th^n the-Power groyip). These ^results were al 1' predictable because 
• the 49 pages c£ the old rules inherently are more cumbersome and 

■ i 

time consuming to use than the H pages of the new rules. The 
Power groups could be expected to work with the old rules faster 
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r ' ■ ■ . • .. 

due to prior experience. The results support the' conclusion^ that 
ihe new rules are more usable for Recreational boaters. 



Pi fficul ty Ratings 



All subjects rated how difficult it was to use the particular 
♦set bf rules that was assigned to them. The rating scale ranged 
from 5 (haVrd rto use) to 1 (easy to use). Figure 5 shows the 
differences in the ratings given by the two groups. 



(Hard) 5 j— 



c 
s 
«p 

> 

u 

it 
, 5 



(Easy) 1 




-• Power 



o- — o Gettysburg 



Old Rules 



New Rules 



Figure 5, The average ratings of how difficult 
it was to use the new and old rules. 



It is clear from the figure that both .experienced and inexperienced \ 
boaters judged the new rules easier to use. This result was 
statistically significant and. supports our interpretation that the 

* » * 

new rules are more usable and more comprehensible . '* 
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Secondary Analysis: Errors 



We conducted an additional analysis of the types of errors 
* 

made'by the different subject groups in an attempt to better 



19 



22 
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understand the differences^ in test performance • The' subjects could 

make three types of esrors on the test: * > 

' f . ' -< ' 

Type 1: The subject could answer a question correctly , yet* 



g rve the wrong rule * 



Type 2: The subject could answer a question ^incorrectly bljt 
give*-the right rule. 



Type 3: The subject could get both the question and the rule 
wrong. 



Table 2 shows the proportion of errors of each type for tJhe 
different subject groups.. For example, of all errors made .by the 
Gettysburg-New gropp, 30% were of the Type 1 (right answer, wrj^ng 
rule), 38% were of the Type 2 (wrong answer, right rule), and 32% 
were of the Type 3 (wrong answer, wrong rule). 



TABLE 2 

Proportion and Type of Errors by Subject Grqups 



Subject 
Grouc 



TYPE 1 ERROR 
f Right Answer 
Wrong Ru*e) 



4 TYPE 2 ERROR 
{Wrong Answer, 
Right Rute) 



Numbers in parent heves jro the <j< \ uM frequencies of each type of error 



TYPE 3 ERROR 
(Wranq Answers 
Wrong Rule) 



Gettysburg Old 


Q6 


(11)* 


19 ' {38} 


75 


(147) 


Poww - Old 


28 


(531 


12 {23} 


60 


(114) 


* Gettysburg New \ 


30 


(24) 


38 (31) 


32 


( 26) 


Power Now* 


36 


(431 


24 (29) 


39 


( 47) ' 



Table 2 illustrates several in teres ting -performance patterns. 
The first is *th£ fc pattern of errors mad t e by subjects who us/?d the 
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old regulations. The Gettysburg-Old subjects made very few of the 
* * 

first type of error (right answer, wrong rule). This indicates * 

[ t\iat these inexperienced subjects really did not know much about 
v 

marine radios. Furthermore, this finding adds support to the 
validity of the test in that subjects were not able to "guess" 
correct answers. In contrast, the Power-Old subjects, who were * 
experienced boaters, were able to reapond correctly, despi te ' using 
the wrong rule." This, too, is an indicator that tVe test asked 
"good" questions because if experienced ma-rine, radio operators / 
could not Bnswer any questions, ,£t would be possible to argue that 
the # test was unrealistic. * . - 

Another interesting pattern shov^n in Table 2 is the large i 
increase in proportion of Type 1 err6r (right answer, wrong rule) ' 
for the Gettysburg-New group as compared to the Gettysburg-Old 
group. A possible interpretation, of this unexpected^ finding is 
that the Gettysburg-New subjects. were able to read through the • - 
entire regulation and thereby increase their general knowledge of 
marine radios. Thus, they were able to answer some. questions ^ 
correctly despite going to the' wrong rule. 

•A final pattern of interest in Table 2 is that Type 2 errors 

, s 

(wrong answer, right rule) still occurred with the new rules. This 
suggests that ther£ are still some problems with the new rules, 
eveawith^all the effort that went into writing -them in plain 
English and in redesigning them. 
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.Discussion * 

We pointed out earlier that'bbth the FCC and the Document 
Design Project ha* two somewhat different goals 'for this 
evaluation. On the one hand, we bot\\ were interested in knowing 
whether government rules that were written- in plain English *and 
that incorporated other document desiqn principles were 
understandable arrt3 usable, and whether the revised rules were any 
etter than 'the original ones. 



/ 



On the other hand f we both were concerned about the degree to 

f 

which non-research trained document designers could carr^ out a 
useful evaluation of a public document. The. FCC document designers 
were looking for evidence that demonstrated that they were capable 
of developing readable rules and regulations and that also could be 
used to convince their , superiors of the value of "document design 
activities. The DDP wanted to monitor the entire process of 
non-evaLuator s conducting an evaluation in order to find out 
whether they cou^d do it and to learn more about how ITo help other 
non-research , trained document designers conduct empirical 
evaluations . * ^ 

Are the Revised Rules Clea'r^ahd U&able? . ' ^ • 

— ; ■ -a 

The statistical evidence generated by/Che 'evaluation on all 
four dependent measures supports the conclusion that subjects using 
the new rules performed far better than subjects using the old 
rules. More questions -were answered correctly, the correct rules 



V 22 2" 



were identified more often, time per question -was faster', > ahd the 
new rules were judged easier to use. Moreover, these effects w$re 
found with both experienced and inexperienced recreational boaters/ 
Even on measures where experienced boaters had an advantage by 
virtue of previous exposure to or knowledge of the -old . regulations , 
the advantage disappeared with the new '^ules. 



.The, evaluation' strongly suggests that the FCC group that 

revised the ^ules met their overall goal. The revised rules are' 

indeed usable, readable, and easy to'^undersband by' the kinds of 

people expected to Qse them., In addition, \hey are superior 'to the 

existing rules. The use of simpler language, distinctive headings, 

* * 

logical presentation, and the el imina tion ,of technical gargon and 
superfluous information does mak'e a*d if f erehce . We do not know 
whether this increased clarity ^scost-ef fective to the government 
in terms of the time and'expense ta^ktfi' to revise the rule. Neither 



do we 'know if ths new rules result Unreduced errors and accidents 
by boaters, or if they resulted in increas&rd good will of citizen 
boaters.. This study did not examine these v issues . Yet, it would 

be ftard to argue against 'the value of a set of government rules 

« ** ****** » 

that- had the. demonstrated effect on users that the FCC ' s 'ma'r ine 

radio rules had. 1 - • . . 
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The implications are clear: government regulations can be 
improved and qari be made more usable and comprehensible for'^heir 
users. Similan results could occur by revising^other cumb^rsorhe 
.public documents, not only within- the FCC but in other* government ' J 
-agencies as well. \ 



I 
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Is the Evaluation Useful? , 

\ s • * 

The answer to\this depends upon the, answers to still further 
* ^ \ . ' 

questions. Useful to whom? Tc* the FCC? To DDP? 

■J ' —- 

From the FCC'S pbin t w of * view the evaluation provides direct 
and consistent evidence that the revised marine radio rules* are 
clear^and usable to experienced and potential recreational boaters. 
-This evidence is independent of any per sonal \opinion about the 
quality of the revised rules or of any personal feelings about the 
value of the plain English movement. What the FCC does with the 
information generated by thisSevaluation is up to the revlsors of 

r 

the marine radio* rules and the appropriate FCC decision makers. As 

r s 

document designers and evaluators, we would hope that the success 

^ 9 * 

of the effort both to rewrite* and to evaluate rules will spur the . 

-FCC and other , government agencies to conduct similar projects. 

From the DDP's point of view the evaluation is a succes-a. We' 
* ♦ 

believe that the experience has shown that noh-research trained 
document designers can\^Ia'n and conduct' a useful empirical 

evaluation of' a public document. We believe thrat typical dcrcum^nt 

^ ■ r 

designers can learn how to carry out all essential i evaluation 

ta a sks, with the possible exception of data analysis. 4 They can 

/learn to identify the^purpose of an evaluation, to articulate the 

vaTue of empirical e valuat ion, to identify the appropriate sample 

of document users ana secure their cooperation, to design suitable 

performance measure^, to implement standardized data collection 



. . . . . 1 ; 48T « • ' 

activities, £nd to interpret the significance of *any findings for 



fu£dre decisions. 



■27 



MMB-^EGIN : 

TIME FINISH, ; 

1. How must ypu identify your station when your ar< 
communicating? ' ^ 

(a) By the name of the vessel. 

(b) By the vessel's state registration or 
documentation ^number. 

(c) By FCC call, sign, * ^ 

(d) By FCC cail sign and boat name. 



" V 

RULE Sgf^ION: \ 
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A-l 



TIME BEjGIN: 
f IME FINISH 



2. When must you identify your station? 
* (a) At "the beginning of a message. 
\bk At the enc^pf a message. , 
(ej At both the beginning and the end of 
message. 

(d)" Whenever you change channels and at 
end of a message. 

fcULE SECTION: 



A. 



30 



TIME BEGIN _ 
TIME FINISH 



3. Do recreational boaters have to keep- a radio 
log? 

Yes or' No (circle one) 



RUL^»'SECTIOI?; 



\ 



9 



A- 3 31 



TIME BEGIN: 
TIME FINISH:* 



4. What rfkust yod enter in a radio log? 



\ 



RULE SECTION: ' > n_ 



f 



A- 4 
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TIME BEGIN: 
TIME FINISH: 



5. Thomas Miller is licensed to operate a marine radio 
aboard his recreational boat. He then sells his 

boat to a neighbor and t£lls hint that the^ship 

' / 
station license goes^with the boat. Is (Mr. Millar 

• » 
correct? * > 

Yes or No (circle one) K 

Why or Why Not? 



RtjLE SECTION : 



J 



/ 



A- 5 33 



TIME BEGIN: 
TIME FINISH: 



6, - If Tljomas Miller's ship station license expired 
yesterday but he has applied for rei^wal, can he 
still operate his marine radio? 
^es or No (circle one) 



r 

RULE SECTION: 



34 



TIME BEGIN: 
TIME FINISH: 



t 

7. You are on a trip with several 'other boats from ' 
your yacht club and wish ( to remain in contact 
with one another. Since everyone's radio is equipped 
with chanjiel 6, you agree to use this channel, 

* Is this permitted? 

Y^s or No (circle one) , 

( How about channel 68? , 

\ / 

Yes or No (circle one) <■ 



RULE SECTION: 
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TIME BEGIN£ 
TIME FINISH: 



8. You are participating in an overnight cruise with 
sever another recreational boats.' Which of the 
following channels should yoii use to talk to one 
another: 

V 

(a) Channel 12. 

(b) Channel ~*8 

(c) Channel 22 

(d) Channel . 6 



RULE SECTION : 



36 
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TIME BEGIN: . : ' 

TIME FINISH: , _: 

• ( 

9. Thomas Miller's marine radio has recently 'broken; 
Is it legal for Mr. Miller to repair his radio 
himself 1 ? 

Yes or No (circle one) 

If No, then, who must repair it? « 



RULE SECTION; 
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TIME BEGIN: 
TIME' FINISH: 



10. Thomas Miller recently decided to" put a marine radi 
on his boat. What FCC license or licenses does he 
need? 



RULE SECTION: 



r 




i < \ 



\ 
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TIME BEGIN: 
TIME FINISH: 



What form or forms should Mr. Miller use to apply 
• for his license or licenses? 




RULE SECTION; 



/ 
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TIME BEGIN: : * 

— ; «o> 



TIME FINISH: 



12. Thomas Mi\Lle;r is operating his recreational boat , 

and sees a floating log which is a hazard to navir- ^. , 
gation. He wishes t to contact the Coast Guard to 
report this hazard. What channel or. channels should 
fie use? 
(a) Channel 22 
. (b) Channel ii 
\c) Channel 6 " ¥ _ ' 

(d) Channel 16 



RULE SECTION; 
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r 



i 
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TIME BEGIN.: - 



TIME FINISH:; 



13*. Which emergency signal would Thoma°s* Miller use 
\t he wished to' warn another vessel of *a 
navigational hazard? "* . 

(a) PAN Pi^f ^ 1 

(b) SECURITY " ^ 
(C) MAYDAY X " 



RULE SECTION: 
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14. Did. you thipk these rules were-^; (check one) 

V^y Difficult to Understand: 1 

Difficult to Understand: , ^ 

Of Average Difficulty: : 

Easy to Understand: 



j Very Easy to Understand: 



V 
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